Skip to content

fix(workflows): load custom steps before add - #4473

Open
ton-katsu wants to merge 1 commit into
github:mainfrom
ton-katsu:fix/workflow-load-custom-steps
Open

fix(workflows): load custom steps before add#4473
ton-katsu wants to merge 1 commit into
github:mainfrom
ton-katsu:fix/workflow-load-custom-steps

Conversation

@ton-katsu

@ton-katsu ton-katsu commented Sep 8, 2026

Copy link
Copy Markdown

Assisted-by: GitHub Copilot (model: unknown, supervised)

Description

Fix the workflow add command so it loads custom workflow steps before
validating a workflow.

Previously, workflow add did not call load_custom_steps(project_root).
As a result, workflows containing custom steps such as feedback could not be
validated and added successfully.

A regression test was added to verify that the project root is resolved before
custom steps are loaded.

Testing

  • Tested locally with uv run specify --help
  • Ran focused tests with .venv/bin/python -m pytest tests/test_workflows.py -q
  • Ran the full test suite with uv sync && uv run pytest
  • Tested with a sample project
  • Verified the regression test on macOS and Linux

Focused test result:

970 passed, 1 skipped

The sample project test verified that:

  • The feedback step was installed from a step catalog.
  • A workflow containing the feedback step was added successfully with
    specify workflow add --dev.

The full test suite was not marked as passing because running uv sync && uv run pytest without the test extra removed the test dependencies and resulted
in collection errors. The focused workflow test suite passed successfully.

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance

The code changes and regression tests were created with GitHub Copilot. The
changes were reviewed and tested by the contributor.

@ton-katsu
ton-katsu requested a review from mnriem as a code owner September 8, 2026 18:25
@mnriem
mnriem requested a balanced review from Copilot September 9, 2026 14:12
@mnriem mnriem added the triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review label Sep 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The targeted fix is correct, consistent with existing workflow commands, and covered by a regression test.

Pull request overview

Loads project-installed workflow steps before validating newly added workflows.

Changes:

  • Loads custom steps after resolving the project root.
  • Adds a regression test for the loading order and root argument.
File summaries
File Description
src/specify_cli/workflows/_commands.py Loads custom steps before workflow validation.
tests/test_workflows.py Verifies custom-step loading uses the resolved project root.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@mnriem mnriem added author-awaiting Waiting on author response author-needs-tests Real change but missing a regression test — add one that fails before / passes after labels Sep 9, 2026
@mnriem

mnriem commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

The fix itself is right (review's green, ubuntu passes), but your new regression test test_add_loads_custom_steps_after_resolving_project fails on macOS and Windows — CI shows AssertionError: assert [PosixPath('/...')] == [PosixPath('/...')] where the paths look identical but compare unequal. That's the macOS /tmp/private/tmp (and Windows short-path) symlink normalization: tempfile paths don't round-trip through .resolve(). Please make the assertion resolve both sides first — e.g. compare [p.resolve() for p in loaded_roots] == [tmp_path.resolve()] (or assert on the resolved project_root). Re-request review once CI is green across all runners

@ton-katsu
ton-katsu force-pushed the fix/workflow-load-custom-steps branch from bfb33d7 to 900042a Compare September 10, 2026 01:44
@ton-katsu

Copy link
Copy Markdown
Author

@mnriem
The test fix is pushed.
I also ran the focused regression test on macOS and Linux, and it passed:

tests/test_workflows.py::TestWorkflowAddCaseInsensitiveSuffix::test_add_loads_custom_steps_after_resolving_project PASSED

The path comparison now resolves both paths before asserting equality.
Could a maintainer please approve or rerun the CI workflow so the Windows runners can verify the fix?

@mnriem
mnriem requested a balanced review from Copilot September 10, 2026 14:11
@mnriem mnriem removed the author-needs-tests Real change but missing a regression test — add one that fails before / passes after label Sep 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The regression test does not verify that a workflow using a custom step validates and installs successfully.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread tests/test_workflows.py Outdated
@mnriem

mnriem commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Thanks @ton-katsu — the macOS/Linux test failure is sorted, appreciated. One more from the latest automated review, and it's a real one: test_add_loads_custom_steps_after_resolving_project is currently tautological — it mocks load_custom_steps to a no-op and uses a built-in-step workflow, so it only proves the loader was called, and would still pass if the fix regressed (e.g. if the load moved after validation). Could you make it a genuine regression: have the workflow reference a custom step, have the loader actually register that step, and assert the workflow adds successfully — such that the test fails if loading happens after validation? The fix itself is right; this just makes the test guard it.

Assisted-by: GitHub Copilot (model: unknown, supervised)
@ton-katsu
ton-katsu force-pushed the fix/workflow-load-custom-steps branch from 900042a to a8e0e46 Compare September 10, 2026 18:13
@ton-katsu

ton-katsu commented Sep 10, 2026

Copy link
Copy Markdown
Author

@mnriem
Thanks for the careful review. I updated the regression test to exercise the actual behavior:

  • It creates a temporary custom step package with step.yml and __init__.py.
  • The workflow references that custom step type.
  • The real load_custom_steps() loader registers the step.
  • The test asserts that workflow add succeeds and installs the workflow.

This means the test fails if custom-step loading is moved after workflow validation. The test passes on Linux and macOS.
I also verified that the test fails when run against the pre-fix version of _commands.py, confirming that it protects against the original regression.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No unresolved review issues remain, and regression coverage was added.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author-awaiting Waiting on author response triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants